Compare voluntary survey with mandatory census at own risk: StatsCan

OTTAWA -- The first pack of data from 2011's National Household Survey comes with the census equivalent of a Surgeon General's warning: make any historical comparisons at your own risk.

Slapped across the back pages of most of the Statistics Canada documents released Wednesday is a disclaimer that the voluntary National Household Survey is an altogether different beast than the now-scrapped mandatory long-form census.

The Harper government touched off a controversy in 2010 when it decided to replace the mandatory long-form census with a voluntary survey. Demographers, analysts and insiders fretted that the quality of the data would suffer, especially when it comes to studying smaller geographic areas -- towns and neighbourhoods for example.

The agency's note to readers would appear to validate some of their concerns.

"The NHS estimates are derived from a voluntary survey and are therefore subject to potentially higher non-response error than those derived from the 2006 census long form," the agency says repeatedly throughout the material released Wednesday.

When it came to comparing the 2006 and 2011 results, the smallest communities had the biggest differences.

The agency says the National Household Survey was most likely to differ from its long-form predecessor in census subdivisions of fewer than 25,000 people. While larger communities were pretty much identical in 2006 and 2011, Statistics Canada says that for smaller areas "there can be large discrepancies between the 2011 census population count and NHS population estimate."

"Those discrepancies are due to weighting and, as in any survey, they may be larger for small geographic areas," Statistics Canada says.

Industry Minister Christian Paradis issued a news release Wednesday, emphasizing that more Canadians filled out the new survey than the previous long-form census.

"Our government is committed to collecting statistical data while protecting Canadians' privacy," the minister said. "And Canadians responded: more than 2.5 million households returned the survey, achieving a response rate of 68 per cent and making this the largest voluntary survey ever conducted in Canada."

While more people filled out the survey because the government increased the proportion of households destined to receive it, the NHS fell far short of the 94 per cent response rate the long-form census was expected to achieve.

Statisticians have pointed out that with a voluntary survey it's not just how many people who fill it out that's important, but who fills it out. Certain groups can be less likely to return the forms -- low-income earners and aboriginals for example. That creates a bias in the data that is tricky to fix.

Perhaps the biggest is a rather glaring footnote in the handout entitled, "Immigration and Ethnocultural Diversity in Canada."

While the National Household Survey showed that most people who came to Canada between 2006 and 2011 were born in the Philippines, that finding didn't jibe with the federal government department that tracks immigration.

"This result was not in line with administrative data from the Department of Citizenship and Immigration Canada which provides the number of recent immigrants by their country of birth settling in Canada each year," the footnote says.

But that wasn't Statistics Canada's only headache.

Past efforts to count First Nations communities have been hampered by the refusal of some groups to participate and their latest attempt was no different.

Another note to readers says 36 of the country's 863 reserves were not counted because census-takers either weren't allowed in or were interrupted before they could finish their work, or because of natural events such as forest fires.

"Most of the people living on reserves are First Nations registered Indians, and consequently, the impact of the incomplete enumeration will be greatest on data for this population," the note says.

Statistics Canada itself has suggested that some important questions on the survey should be included in the still-mandatory short census, in order to ensure reliable data.

Paradis indicated in his statement that the government will look at ways to enhance the process.

"This was the first time a voluntary National Household Survey was undertaken," Paradis said. "Our government will be looking at options to improve the quality and reliability of the data generated by the 2016 census cycle."